Training and/or utilizing a model for predicting measures reflecting both quality and popularity of content

ABSTRACT

Implementations relate to training a model that can be used to process values for defined features, where the values are specific to a user account, to generate a predicted user measure that reflects both popularity and quality of the user account. The model is trained based on losses that are each generated as a function of both a corresponding generated popularity measure and a corresponding generated quality measure of a corresponding training instance. Accordingly, the model can be trained to generate, based on values for a given user account, a single measure that reflects both quality and popularity of the given user account. Implementations are additionally or alternatively directed to utilizing such predicted user measures to restrict provisioning of content items that are from user accounts having respective predicted user measures that fail to satisfy a threshold.

BACKGROUND

Online platforms exist that enable creation of user accounts for theonline platform, and that enable utilization of the user accounts topublish content items to the online platform under the name of the useraccount. For example, a user account can be used to craft an originalcontent item that includes text, image(s), video(s), and/or othercontent. The content item can be published to the online platform andascribed, by the online platform, as generated by the user account. Asanother example, a user account can be utilized to share a content itemthat was crafted by another user account of the online platform. Thecontent item can be ascribed, by the online platform, as generated(e.g., shared) by the user account, but generated by another useraccount. User accounts of an online platform can additionally oralternatively be utilized in publishing content items to other onlineplatforms or, more generally, to the Internet. As one example, a useraccount of a first online platform can be utilized to sign-in to asecond online platform. Content generated by the user account can bepublished to the second online platform and ascribed by the secondonline platform and based on the sign-in with the user account, asgenerated by the user account. As another example, a user can cause acontent item to be published to the Internet and independent of anonline platform. In causing the content item to be published, the usercan ascribe to the content item, user account information thatexplicitly or implicitly identifies a user account of the user, such asa user account of an online platform.

SUMMARY

Some implementations of this specification are directed to training amodel (e.g., a regression model or other machine learning model) thatcan be used to process values for defined features, where the values arespecific to a user account. The values can be processed using the modelto generate a predicted user account measure (also referred to herein asa “predicted measure” or “predicted user measure”) that reflects bothpopularity and quality of the user account.

In implementations that are directed to training the model, traininginstances are generated that each include training instance input ofvalues that are specific to a corresponding user account, such as a useraccount of an online platform. The training instances each furtherinclude labeled training instance outputs that include a correspondinggenerated quality measure for the user account, as well as acorresponding generated popularity measure for the user account. In someimplementations, the quality measure for the user account can begenerated by providing an online account page for the user account(e.g., the home page for the user account on the corresponding onlineplatform) for a plurality of discrete quality evaluations (e.g., bycorresponding human reviewers utilizing respective client devices),receiving the discrete quality evaluations (e.g., respective ratings ona rating scale), and generating the quality measure as a function of thediscrete quality evaluations. In some implementations, the popularitymeasure for the user account can be generated as a function of aquantity of end-user interactions with the account page for the useraccount, as determined based on historical records. For example, thepopularity measure can be the greater of (a) one (or other fixed value)or (b) the logarithm of the quantity of interactions with the onlineaccount page by a population of users and over a time period (e.g., thelast 3 months). For instance, the quantity of interactions can be aquantity of visits to the account page as determined from selections ofsearch result(s) for the search page and/or web browser historical data.

Further, in implementations that are directed to training the model, themodel is trained based on losses that are each generated as a functionof both the popularity measure and the quality measure of acorresponding training instance. Put another way, the model is trainedbased on multiple objectives, where the multiple objectives include bothquality (as reflected by the quality measure labeled outputs) andpopularity (as reflected by the popularity measure labeled outputs). Inthese and other manners, the model can be trained to generate, based onvalues for a given user account, a single user account measure thatreflects both quality and popularity of the given user account.

As described herein, such measures can be used to restrict provisioningof content items that are from user accounts having respective measuresthat fail to satisfy a threshold. This can prevent content items fromlow quality and/or low popularity accounts from being retrieved and/orcan prevent such content items from being transmitted to and/or renderedat client devices, thereby conserving various associated computationaland/or network resources. Further, this can additionally and/oralternatively promote retrieving, transmitting, and rendering of contentitems from high quality and/or high popularity accounts. This canincrease the probability that users to whom the content items arerendered will engage with such content items and/or that such contentitems will satisfy the informational needs of the users, therebymitigating the likelihood that such users will perform furthercomputational activities (e.g., further searching, further browsing) tofulfill those informational needs. This can additionally and/oralternatively decrease the probability that such content items willinclude low-quality content that could be harmful to users (e.g.,obscenities or other offensive content) and/or to corresponding clientdevices (e.g., nefarious links or other content).

In some of the implementations that are directed to training the model,the loss that is generated based on both a quality measure and apopularity measure of a labeled output of a training instance, can be afunction of a first error (e.g., a mean squared error or other error)and a second error (e.g., a mean squared error or other error). Thefirst error can be determined based on comparing a predicted measure(generated by processing input of the training instance using thecurrent model) to the quality measure. The second error can bedetermined based on comparing the predicted measure to the popularitymeasure. In some of those implementations, the first and second errorscan each be weighted with respective weightings. In some of thoseimplementations, weights can be determined using black box optimization.Optionally, the first weighting for the first error can be the weightdetermined using Pareto optimization and the second weighting for thesecond error can be one minus the weight, or vice versa.

In some versions of those implementations, multiple models can betrained, each utilizing a unique combination of a first weighting forthe first error and a second weighting for the second error. In theversions that train multiple models each utilizing a unique combinationof the first weighting and the second weighting, a subset (e.g., one) ofthe models can then be selected for deployment/use based on evaluatingthe model(s). Evaluating each of the models can be in view of one ormore metric(s) that seek to evaluate whether the model effectivelygenerates predicted measures that discriminate between high qualityand/or high popularity user accounts and low quality and/or lowpopularity user accounts. For example, a model can be evaluated based ona quality metric. For instance, the quality metric can be based on howmany known low quality user accounts, when their corresponding valuesare processed using the model, result in predicted measures that fail tosatisfy a threshold (with higher quantities indicating the model iseffectively filtering out low quality user accounts). As anotherexample, the model can additionally or alternatively be evaluated basedon a popularity metric. For instance, the popularity metric can be basedon how many known highly popular user accounts, when their correspondingvalues are processed using the model, result in predicted measures thatsatisfy the threshold (with higher quantities indicating the model iseffectively including highly popular user accounts). In these and othermanners, multiple models can be trained, but only model(s) that satisfythe further evaluation actually deployed. This can further ensure thatcontent items from low quality and/or low popularity user accounts arerestricted based on measures generated using the deployed model, whilecontent items from high quality and/or high popularity user accounts areutilized—and resulting technical benefit(s) are achieved.

Which metric(s) (e.g. popularity metric, quality metric, and/or othermetric(s)) are utilized in an evaluation, and/or how such metrics areutilized in the evaluation, can be based on one or more considerations.For example, the consideration(s) can include desired performance ofproduct(s) in which a selected model will be deployed and/or otherconsiderations. As one particular example of utilizing metrics, whenmultiple models are trained a given one of those models can be selectedbased on the given one of the models having a quality metric thatsatisfies a threshold and based on having a popularity metric that ishighest amongst the popularity metric(s) of any other models that alsohave respective quality metrics that satisfy the threshold. As anotherparticular example, assume that a model is already deployed. Thedeployed model can be one that is also trained based on losses generatedbased on both a quality measure and a popularity measure, or can be amodel trained utilizing alternate techniques. In such an example,popularity and quality metrics can be generated for the deployed model,and popularity and quality metrics can also be generated for a givenmodel trained based on losses generated based on both a quality measureand a popularity measure. If the quality metric for the given model isgreater than or equal to the quality metric for the deployed model, andthe popularity metric for the given model is greater than the popularitymetric for the deployed model, the given model can be selected forreplacing the deployed model. Otherwise, the given model will not beselected for replacing the deployed model.

Some implementations are additionally or alternatively directed togenerating, using a trained model, corresponding user account measuresfor each of a plurality of user accounts, such as user accounts of anonline platform. For example, for a given user account, values fordefined features of the trained model can be identified, where thevalues are specific to the given user account. Those values can beprocessed using the trained model to generate a predicted user measurefor the user account. The predicted user measure and its association tothe user account can then be stored and utilized for one or morepurposes.

For example, some implementations are additionally or alternativelydirected to utilizing the generated predicted user measures, for theuser accounts, in determining whether to render content items generatedby (e.g., crafted by or shared by) the user accounts. In some of thoseimplementations, a search of content items (e.g., posts or other contentitem(s) from an online platform) can be performed based on one or morequeries to identify content items that are responsive to the one or morequeries. A content item can be responsive to the one or more queriesbased on content including text, image(s), and/or other content thatmatches term(s) or other parameter(s) of the one or more queries. Thepredicted user measures can be used to restrict the search to onlycontent items that are from those user account(s) whose predicted usermeasures satisfy a threshold. This can result in a more efficient (e.g.,less computational time and/or faster) search by reducing the corpus ofcontent items that are searched. Further, this can also preventtransmitting and/or rendering of any content items from user accountswhose predicted user measures fail to satisfy the threshold (as they arenot searched). Additionally or alternatively, the predicted usermeasures can be used to filter out any responsive content items that arefrom the user account(s) whose predicted user measures fail to satisfy athreshold. This can prevent wasteful transmission of such content itemsto a client device and/or rendering of such content items at a clientdevice, as they are filtered out based on the predicted user measures.This can additionally and/or alternatively decrease the probability thatsuch content items will include low-quality content that could beharmful to users (e.g., obscenities or other offensive content) and/orto corresponding client devices (e.g., nefarious links or othercontent).

In some implementations that search content items (e.g., from an onlineplatform) based on a query, the search is performed responsive to asubmission of the query by a client device. In those implementations,the content items that are transmitted and caused to be rendered can betransmitted responsive to the submission of the query and presented assome of the search results that are responsive to the query. Optionally,other types of search results can also be identified, transmitted, andrendered responsive to the query. For example, if the content itemssearched are from an online platform are all posts, search results thatare not posts (or links to posts) and that are not from the onlineplatform can also be identified, transmitted, and rendered responsive tothe query. For instance, search results can be provided for webpagesand/or images that are not from the online platform.

In some implementations that search content items based on a query, thecontent items that are transmitted and caused to be rendered at a clientdevice can be transmitted and caused to be rendered at the client deviceindependent of submission of the query at the client device. Forexample, they can be transmitted as a push notification that isproactively rendered at the client device. For instance, the query canbe identified as relevant to the client device based on a location ofthe client device and/or interests of a user of the client device, andthe content items provided in a push notification based on the querybeing identified as relevant to the client device. In some versions ofthose implementations, the search based on the query is performed basedon determining that the query, and/or related query/queries, aretrending (e.g., experiencing an uptick in submission rate) in one ormore geographic areas.

As described herein, some implementations that utilize generatedpredicted user measures, for user accounts, in determining whether torender content items published by the user accounts, compare thepredicted user measures to a threshold. If the predicted user measuresfail to satisfy the threshold, content items from the corresponding useraccounts can be restricted. In some of those implementations, thethreshold is fixed across multiple (or even all) types of queries. Inother implementations, the threshold can be dynamically determined basedon one or more properties of the query. For example, the threshold canbe determined based on one or more classifications of the query, such asa primary (e.g., strongest) classification. For instance, where highermeasures indicate higher popularity and/or higher quality user accounts,the threshold for “sports” queries can be lower than the threshold for“political” queries (i.e., the bar is higher for “political” queriesthan it is for “sports” queries). Also for instance, the threshold for“food” queries can be lower than the threshold for “news” queries. Asanother example, the threshold can be determined based on an extent oftrending of the query, such as a geographical extent of trending of thequery. For instance, where higher measures indicate higher popularityand/or higher quality user accounts, the threshold for a query that istrending only in a limited quantity (e.g., 10 or less) of cities can belower than the threshold for a query that is trending nationally or evenglobally (i.e., the bar is higher for queries with a large geographicextent of trending). Also for instance, the threshold for a query thatis trending to a first degree that is lesser than a second degree towhich an additional query is trending, can be lower than the thresholdfor the additional query.

As described herein, in training and/or utilizing a model, values forfeatures of user accounts are identified and processed using the model.Various features can be utilized. Some non-limiting examples of suchfeatures include: a Pagerank measure for an online account page for theuser account; a quantity of user interactions with the online accountpage; a status of the user account as assigned by the online platform, asentiment measure that is based on content items that are generated bythe user account; a quantity of links, to the online account page, fromdomains that are in addition to online platform domains associated withthe online platform for the user account; a link quality measure that isbased on links included in content items that are generated by the useraccount; and/or a primary language for the user account. It is notedthat, in some implementations, different models are utilized fordifferent geographic regions and/or different languages. In some ofthose implementations, those different models can utilize different setsof features. For example, an English language based model may utilize aset of features, and a non-English language(s) based model may utilizeonly a subset of that set of features. As described herein, in someimplementations training instances used in training the non-Englishlanguage(s) based model can still be based on English language basedtraining instances (e.g., by removing value(s) for those feature(s) notutilized in the non-English language(s) based model.

In various implementations, predicted user account measures aregenerated offline relative to their usage in determining whether torender content items from corresponding user accounts. Put another way,the predicted user account measures can be generated prior to beingutilized and do not have to be generated online for each newutilization. Rather, new predicted user account measures can begenerated at periodic (e.g., weekly) or non-periodic intervals. In someof those implementations, the predicted user account measures arepre-indexed with (or otherwise assigned to) their corresponding useraccounts. This enables fast and efficient retrieval of the predicteduser account measures, and fast and efficient utilization of themeasures in determining whether to render content items fromcorresponding user accounts. This can reduce response time of, and/orcomputational burden on, computing device(s) in determining whether torender content items from corresponding user accounts.

The above description is provided as an overview of variousimplementations. The below description provides additional detail forthose implementations, and for various additional implementations.

It should be appreciated that all combinations of the foregoing conceptsand additional concepts described in greater detail herein arecontemplated as being part of the subject matter disclosed herein. Forexample, all combinations of claimed subject matter appearing at the endof this disclosure are contemplated as being part of the subject matterdisclosed herein.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example environment in whichimplementations described herein can be implemented.

FIG. 2 is a flowchart illustrating an example method of generatingtraining instances, for use in training a model for generating predicteduser measures, according to various implementations disclosed herein.

FIG. 3 is a flowchart illustrating an example method of trainingmodel(s) for generating predicted user measures, using both labeledquality measures and labeled popularity measures, according to variousimplementations disclosed herein.

FIG. 4 is a flowchart illustrating an example method of using a trainedmodel in generating user account measures for user accounts, accordingto various implementations disclosed herein.

FIG. 5 is a flowchart illustrating an example method of using useraccount measures for user accounts, according to various implementationsdisclosed herein.

FIG. 6A and FIG. 6B each illustrates an example graphical user interfacefor presenting content items, selected based at least in part on useraccount measures, at a client device.

FIG. 7 illustrates an example architecture of a computing device.

DETAILED DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an example environment in whichimplementations described herein can be implemented. Although notillustrated in FIG. 1 , the example environment can include one or morecommunication networks that facilitate communication between variouscomponents and/or subcomponents in the environment (e.g., via networkinterfaces of those components). Such communication network(s) mayinclude a wide area network (WAN) such as the Internet, one or moreintranets, and/or one or more bus subsystems—and may optionally utilizeone or more standard communications technologies, protocols, and/orinter-process communication techniques.

FIG. 1 includes a training instance system 130 that can generatetraining instances that are each based on a user account, such as a useraccount of a given one of one or more online platforms 110, and storestraining instances in training instances database 156. FIG. 1 alsoincludes a training system 140 that utilizes the training instances, oftraining instances database 156, to train one or more models 158 togenerate predicted user account measures by processing correspondingvalues for defined features. Further, FIG. 1 includes a provisioningsystem 160 that can utilize model(s) 158 to generate predicted useraccount measures, and use such measures in determining which contentitems are retrieved and/or transmitted to client device(s) 114. Forexample, provisioning system 160 can utilize such measures indetermining to provide, to one of the client device(s) 114 proactivelyor in response to a request, only responsive content item(s) that arefrom user account(s) having associated measures that satisfy athreshold.

Training instance system 130 is illustrated with a quality measureengine 132, a popularity measure engine 134, and an input feature valuesengine 136. The training system 130 can select, from online platform(s)110, a subset of user accounts for which to generate training instancesfor storing in training instances database 156. For example, stratifiedsampling, uniform random sampling, and/or other sampling techniques canbe utilized to select a subset of user accounts. For example, a givenone of online platform(s) 110 can be a microblogging and social networkplatform on which users post and interact with content items that areelectronic messages or posts. Such an online platform can includemillions of user accounts, and the training system 130 can select 1,000,2,000, or other subset of those user accounts. For each of the selecteduser accounts, the quality measure engine 132 generates a qualitymeasure for the user account, the popularity measure engine 134generates a popularity measure for the user account, and the inputfeature values engine 136 identifies values for defined features, wherethe values are specific to the user account. The training instancesystem 130 generates a training instance based on each user account,where the training instance includes input that includes the valuesidentified by input feature values engine 134 for that user account, andincludes labeled outputs that include the quality measure and thepopularity measure generated by engines 132 and 134 for that useraccount.

In generating a quality measure for a user account, the quality measureengine 132 can interface with one or more quality evaluators 112. Thequality measure engine 132 can provide, to the one or more qualityevaluators 112, information related to the user account. The qualityevaluator(s) 112 can each respond with a corresponding discrete qualityevaluation, such as a discrete quality evaluation that is a rating on arating scale. For example, the rating scale can be a continuous scalewith options from 0 to 1 for quality, with 1 being the highest qualityand 0 being the lowest quality. The quality measure engine 132 can thengenerate the quality measure, for the user account, as a function ofreceived discrete quality evaluations. For example, the quality measurecan be based on the minimum of all received quality measures, as thenext-to-minimum (i.e., the second lowest) of all received qualitymeasures, or as an average of all received quality measures.

In some implementations, the information related to the user account,that is provided to the quality evaluators, includes an online accountpage (e.g., a link to the online account page) for the user account(e.g., the home page for the user account on the corresponding onlineplatform). The information can be provided with instructions onevaluating the quality, with an explanation of the rating scale, and/orwith interface elements that can be interacted with to select a discreterating. In some implementations, the quality evaluators 112 are each acorresponding client device operated by a corresponding human, and thediscrete quality evaluations are based on user interface input (e.g.,directed to provided interface elements) provided by the correspondinghuman after reviewing the provided information.

In generating a quality measure for a user account, the popularitymeasure engine 134 can interface with historical records database 152,which can include historical records for the user account. In someimplementations, the historical records database 152 includes datadefining, for the user account, a quantity of user interactions for theuser account, such as a quantity of interactions with the account pagefor the user account. In some of those implementations, the popularitymeasure engine 134 generates the popularity measure for the user accountas a function of the quantity of user interactions. For example, thepopularity measure engine 134 can generate a popularity measure that isthe greater of one (or other fixed value) or the logarithm of thequantity of interactions with the account page. The quantity ofinteractions can be by a population of users and over a time period(e.g., the last month), and can be determined in various manners. Forinstance, the quantity of interactions can be a quantity of visits tothe account page as determined from selections of search result(s) forthe search page and/or web browser historical data.

In generating values, for defined features, for a user account, theinput feature values engine 136 can determine those values utilizinghistorical records database 152, one or more feature models 154, and/orutilizing platform content items 150 for the user account. The inputfeature values engine 136 can determine values for various features,where the values are specific to the user account.

One example feature is a Pagerank measure for an online account page forthe user account. The input feature values engine 136 can determine thevalue of the Pagerank measure, for the user account, based onutilization of the Pagerank algorithm for the online account page forthe user account. Another example feature is a quantity of userinteractions, over a time period, with the online account page, such asthe logarithm of the quantity of user interactions. The input featurevalues engine 136 can determine the value for such a feature, for theuser account, based on taking the logarithm of the quantity of userinteractions. The quantity of user interactions can be identified fromhistorical records database 152. Another example feature is based on astatus of the user account, where the status is assigned by the onlineplatform based on one or more criteria utilized by the online platform.For instance, the status of the user can be either “verified” or “notverified”, and the input feature values engine 136 can assign a value of“1” for the feature if the user account is verified and a value of “0”otherwise. The input feature values engine 136 can determine whether theuser account is verified based on analysis of the account page and/orthrough interaction with an API of the online platform.

Yet another example feature is a sentiment measure that is based oncontent items that are generated by the user account. The input featurevalues engine 136 can determine the value for such a feature, for theuser account, based on processing one or more of the platform contentitems 150, for the user account, using sentiment model(s) of the featuremodel(s) 154. For example, text from such a content item can beprocessed using sentiment model(s) to determine direction(s) (e.g.,positive or negative) and/or magnitude (e.g., how positive or hownegative) of sentiment of the content item. The value for the useraccount can be based on an average, mean, or other function ofdetermined directions and/or magnitudes of the sentiments for multiplecontent items for the user account. For example, ten or other quantitiesof content items for the user account can be sampled, sentimentdirection(s) and/or magnitude(s) determined for each, and a value forthe sentiment determined as a function of all ten direction(s) and/ormagnitude(s).

Yet another example feature is a quantity of links to the online accountpage, from domains that are in addition to online platform domainsassociated with the online platform for the user account. The inputfeature values engine 136 can determine the value for such a feature,for the user account, based on link analysis from crawling the Internet.Yet another example feature is a link quality measure that is based onquality measures for the underlying content in links included in contentitems for the user account. The input feature values engine 136 candetermine the value for such a feature, for the user account, based onidentifying links in platform content items for the user account, usingfeatures model(s) to generate quality measures for the underlyingcontent items in those links, and determining the value as a function ofthe generated quality measures. Yet another example feature is a primarylanguage for the user account. The input feature values engine 136 candetermine the value for such a feature (e.g., 1 for English 0 otherwise;or 1 for English, 2 for French, 3 for Spanish . . . etc.), for the useraccount, based on analysis of the account page and/or content items ofthe user account, and/or through interaction with an API of the onlineplatform. Although various example features have been provided,additional and/or alternative features can be utilized in techniquesdescribed herein.

Turning now to the training system 140, it utilizes training instances,from training instances database 156, to train one or more model(s) 158.The training system 140 trains the model(s) 158 based on losses that areeach generated as a function of both the popularity measure and thequality measure of a corresponding training instance. In someimplementations, where multiple models 158 are trained, each model canbe trained based on the same (or at least overlapping) traininginstances, but each can be trained using a different weighting for theerrors that are based on the popularity measures, in generating thelosses. By using different weightings, different losses are determinedand, as a result, the parameters of the models 158, once trained, aredifferent. In those implementations, the trained models 158 can beevaluated and a subset (e.g., one) selected for deployment based on theevaluation.

In some implementations, the model(s) 158 are each a correspondingregression model with the same architecture, but with differentparameter(s) through training. For example, each of the model(s) 158 canbe a regression model that includes a corresponding univariate function(e.g., piecewise linear curve) for each of the features, and thatgenerates a predicted measure as a function (e.g., additive ormultiplicative) of all of the univariate functions as applied tocorresponding values. For example, a first univariate function canprocess a value for a Pagerank feature, a second univariate function canprocess a value for a sentiment feature, etc. In those implementations,the univariate functions each include corresponding parameter(s)value(s) in the function) that are trained based on the determinedmulti-objective losses. Additional and/or alternative machine learningmodels can optionally be utilized. For example, a regression model thatincludes multivariate functions can be utilized. As another example, afeed forward neural network model with hidden layer(s) can be utilized.

In FIG. 1 the training system 140 is illustrated with an optimalweight(s) engine 142, a training engine 144 that includes amulti-objective loss module 145, and a model evaluation engine 146.

In various implementations, the multi-objective loss module 145(described below), in generating a loss, generates the loss based onboth a quality measure and a popularity measure of a labeled output of atraining instance. The loss can be generated as a function of a firsterror (e.g., a mean squared error (MSE)) and a second error (e.g., anMSE). The first error can be determined by comparing a predicted measure(generated by processing input of the training instance using thecurrent model) to the quality measure. The second error can bedetermined based on comparing the predicted measure to the popularitymeasure. In some of those implementations, the first and second errorscan each be weighted differently during training of a given one of themodels 158. As a particular example, the loss function can berepresented as λ*MSE(quality measure of labeledoutput)+(1−λ)*MSE(popularity measure of labeled output), where λ is aweight between 0 and 1. In the particular example, the weighting of thefirst error is λ, and the weighting of the second error is (1−λ).

The optimal weight(s) engine 142 can be used to determine a plurality ofoptimal values for utilizing the weight in training the model(s) 158.For example, the optimal weight(s) engine 142 can determine optimalvalue(s) for the weight(s) using optimization (e.g., using a black-boxoptimizer). For instance, the optimal weight(s) engine 142 can determinean optimal value using multi-objective/Pareto optimization. The optimalvalue can be used as the weight, and one of the first weighting and thesecond weighting can be the weight, and the other of the first weightingand the second weighting can be one minus that weight. The firstweighting and the second weighting can then be used in generating lossesfor training a corresponding one of the models 158 as described herein.The optimal weight(s) engine 142 can likewise be utilized to generateadditional value(s), each of which can be used in determining weightingsfor use in training an additional of the models 158.

The training engine 144 utilizes the training instances 156 in trainingthe model(s) 158. In training the models 158, the multi-objective lossmodule 145 of the training engine 144 generates losses based oncomparing predicted measures generated from applying inputs of atraining instance to the model, to both a quality measure and apopularity measure of a labeled output of the training instance. Thetraining engine 144 updates parameters of the models 158 based on thelosses. Further, in implementations where multiple model(s) 158 havingthe same architecture are trained, the multi-objective loss module 145can, in training each model 158, use at least a different weight (asdetermined by the optimal weights engine 142) in training the model 158.For example, a first weight can be used in generating losses in traininga first of the models 158, a second weight can be used in generatinglosses in training a second of the models 158, a third weight can beused in generating losses in training a third of the models 158, and soforth.

The model evaluation engine 146 can, when multiple models 158 aretrained (e.g., each utilizing a unique weight in determining losses),evaluate each of the multiple models 158. Further, the model evaluationengine 146 can select a subset of the multiple models 158 fordeployment/use by the provisioning system. The model evaluation engine146 can evaluate each of the models in view of one or more metric(s)that seek to evaluate whether the model effectively generates predictedmeasures that discriminate between high quality and/or high popularityuser accounts and low quality and/or low popularity user accounts. Forexample, the model evaluation engine 146 can evaluate a model based onhow many known low quality user accounts, when their correspondingvalues are processed using the model, result in predicted measures thatfail to satisfy a threshold. As another example, the model evaluationengine 146 can additionally or alternatively evaluate the model based onhow many known highly popular user accounts, when their correspondingvalues are processed using the model, result in predicted measures thatsatisfy the threshold.

Turning now to the provisioning system 160, it is illustrated with auser accounts measures engine 162 that includes a features values module163, a threshold engine 164, a content retrieval engine 166, and anoutput engine 168.

The user accounts measures engine 162 utilizes one of the trainedmodel(s) 158 (e.g., one selected by the model evaluation engine 146) togenerate, for each of a plurality of user accounts of an onlineplatform, a corresponding predicted user account measure. The useraccounts measures engine 162 can store associations of the user accountsto the corresponding generated measures in index 159 and/or otherdatabase(s). The user accounts, for which predicted user accountmeasures are generated and associated, can include those used ingenerating training instances and/or additional user accounts (e.g., alluser accounts of an online platform). In generating a predicted useraccount measure for a user account, the features values module 163 canidentify the values, for those features, that are specific to the useraccount. Feature values module 163 can share one or more (e.g., all)aspects in common with input feature values engine 136 of traininginstance system 130 (and can even be the same module/engine).Accordingly, in identifying the values for features for a user account,feature values module 163 can likewise interface with the onlineplatform 110, the historical records 152, and/or the feature model(s)154.

The content retrieval engine 166 searches platform content items 150,based on one or more queries, to identify content items of onlineplatform(s) 110 that are responsive to the one or more queries. Thecontent retrieval engine 166 can search the platform content items 150directly and/or utilizing an index of the content items, such as aseparate index that is created based on crawling of platform contentitems 150.

In some implementations, the content retrieval engine 166 searches allcontent items 150 (directly or via an index). In other implementations,the content retrieval engine 166 searches only a subset of the contentitems 150. In some of those other implementations, the content itemsthat are not searched by the content retrieval engine 166 can be notsearched based on being from user accounts whose predicted measures(e.g., stored in index 159) fail to satisfy a threshold. In someversions of those implementations, the threshold is a fixed threshold.In some other version of those implementations, the threshold is adynamic threshold. The threshold engine 164 can determine the value ofthe dynamic threshold based on one or more properties of thequery/queries upon which the search is conducted. For example, thethreshold can be based on classification(s) of the query/queries, basedon whether the queries are considered local queries (e.g., include localpoint(s) of interest, are historically more frequent in a local area),and/or based on a geographic extent of current trending of the queries.

In some implementations, the output engine 168 determines which of thecontent item(s) retrieved by content retrieval engine 166 to transmit toone or more of the client device(s) for rendering. In some of thoseimplementations, the content retrieval engine 166 prevents certaincontent item(s) from being transmitted based on those content item(s)being from user account(s) whose corresponding predicted measure failsto satisfy a threshold. For example, the content retrieval engine 166can filter out certain content items that are from user account(s) whosecorresponding predicted measure fails to satisfy a threshold. In some ofthose implementations, the threshold is a fixed threshold. In some otherversion of those implementations, the threshold is a dynamic thresholddetermined by threshold engine 164. In some additional and/oralternative implementations, only a given quantity of content items aretransmitted for rendering at least initially. In some of thoseimplementations, the output engine 168 selects the content items toinclude in that given quantity based at least in part on the predictedaccount measures for user accounts corresponding to the content items.For example, if only three content items are to be transmitted, theoutput engine 168 can select three content items based on them beingfrom the three user accounts with the best predicted user accountmeasures. Optionally, the output engine 168 can also considerquery-dependent and/or content specific measure(s) in selecting thecontent items. For example, in determining whether to select a contentitem, the output engine 168 can consider the predicted account measurefor the corresponding user account, as well as query-dependentmeasure(s) that reflect how closely the content item matches thecorresponding query or queries and/or content specific measure(s).Content specific measure(s) are based on the content item itself(independent of the user account) and can include, for example,popularity of the content item, sentiment of the content item, age ofthe content item, and/or other feature(s) of the content item.

In some additional and/or alternative implementations, the output engine168 utilizes the predicted measures in determining how transmittedcontent items should be rendered (e.g., positional order for rendering)by the client device(s). For example, content item(s) from useraccount(s) with the best predicted user account measure(s) can be causedto be presented positionally higher in a vertical listing, orpositionally left in a horizontal listing, and/or initially in ascrollable carousel listing. Optionally, the output engine 168 can alsoconsider query-dependent and/or content specific measure(s) indetermining how transmitted content items should be rendered.

Turning now to FIG. 2 , a flowchart is provided that illustrates anexample method 200 of generating training instances, for use in traininga model for generating predicted user measures. For convenience, theoperations of the flow chart are described with reference to a systemthat performs the operations. This system may include one or morecomponents, such as one or more computing devices implementing traininginstance system 130 of FIG. 1 . While operations of method 200 are shownin a particular order, this is not meant to be limiting. One or moreoperations may be reordered, omitted or added.

At block 252, the system identifies a user account of an onlineplatform. For example, the system can identify the user account based onsampling of user accounts of the online platform.

At block 254, the system generates a quality measure for the useraccount. For example, the system can generate the quality measure as afunction of multiple discrete quality evaluations. The multiple discretequality evaluations can optionally each be received from a correspondingclient device and be based on user interface input at the client deviceby a corresponding human evaluator. The multiple discrete qualityevaluations can optionally be received in response to the systemtransmitting, to the client devices, information for the user account(e.g., the home page for the user account on the corresponding onlineplatform), optionally along with evaluation instructions and/or userinterface elements for providing the quality evaluations.

At block 256, the system generates a popularity measure for the useraccount. For example, the system can generate the popularity measure asa function of a quantity of user interactions with the account page forthe user account, as determined based on historical records.

At block 258, the system identifies, for defined features, values thatare specific to the user account. For example, the system can identifythe features utilizing historical records, feature model(s), and/orthrough analysis of account page(s) of the user account and/or contentitem(s) of the user account.

At block 260, the system generates and stores a training instance thatincludes the values as input and the quality measure and popularitymeasure as labeled outputs.

At block 262, the system determines whether to identify and generate atraining instance based on an additional user account. If so, the systemproceeds back to block 252 and identifies an additional user account. Ifnot, method 200 ends.

FIG. 3 is a flowchart illustrating a method 300 of training model(s) forgenerating predicted user measures, using both labeled quality measuresand labeled popularity measures. For convenience, the operations of theflow chart are described with reference to a system that performs theoperations. This system may include one or more components, such as oneor more computing devices implementing scoring training system 140 ofFIG. 1 . While operations of method 300 are shown in a particular order,this is not meant to be limiting. One or more operations may bereordered, omitted or added.

At block 352, the system initializes a model to generate a predicteduser account measure based on processing values, for defined features,that are specific to the user account. For example, the system caninitialize a regression model with input(s) for each of the definedfeatures. For instance, each of the inputs can be to a correspondingunivariate function for each of the defined features. Block 352optionally includes sub-block 353, in which the model is initializedwith features that are specific to a geographic locale and/or alanguage. For example, in iterations of method 300 that are for anEnglish language model, the system can initialize the model with inputsfor 10 defined features. On the other hand, for a non-English languagemodel, the system can initialize the model with only 8 of those 10defined features.

At block 354, the system generates, using optimization (e.g., using ablack-box optimizer), a weight. For example, the system can generate theweight using multi-objective/Pareto optimization.

At block 356, the system selects a training instance, such as a traininginstance generated using method 200 of FIG. 2 . Block 356 optionallyincludes sub-block 357. In sub-block 357, if the selected traininginstance is for a language and/or a geographic locale that differs fromthe locale and/or language for which the model was initialized in block352, the system can select, for the training instance, a subset of thevalues of the input—where the selected subset are for those featuresthat are specific to the locale and/or language for which the model wasinitialized. For example, if the initialized model is initialized withonly 8 features, but the training instance includes those 8 and 3 more,the 3 extra can be effectively removed from the training instance.

At block 358, the system processes, using the model, the values of theinput of the selected training instance to generate a predicted useraccount measure.

At block 360, the system determines a loss based on evaluating thepredicted user account measure (generated at block 358) based on thelabeled outputs of the training instance, including both the qualitymeasure and the popularity measure of the labeled outputs. Block 360optionally includes sub-block 361 where, in determining the loss, thesystem uses the weight generated at block 354. For example, the systemcan apply the weight as a weighting to a first error that is based onthe quality measure, and can apply one minus the weight as a weightingto a second error that is based on the popularity measure.

At block 362, the system updates the model based on the loss. Forexample, the system can update parameter(s) of the function(s) for theinput features, when the model is a regression model.

At block 364, the system determines whether to perform additionaltraining of the current model. For example, the system can determine toperform more training based on whether unprocessed training instancesremain, based on whether a threshold quantity of training epochs havebeen performed, based on whether a threshold duration of training hasbeen performed, and/or based on other factor(s). When the decision atblock 364 is yes, the system proceeds back to block 356 and selectsanother training instance.

When the decision at block 364 is no, the system proceeds to block 366and determines whether to initialize and train an additional modelutilizing an additional weight. For example, the system can determine toinitialize and train an additional model when it is determined that athreshold quantity of models trained based on different weights have notyet been trained, when additional not yet utilized optimal weight(s)remain, and/or based on other factor(s).

When the decision at block 366 is yes, the system can proceed back toblock 352, initialize a new model, generate an additional weight atblock 354, and train the new model through multiple iterations of blocks356, 358, 360, and 362 and using the additional weight. It is notedthat, in various implementations, multiple models can be trained, eachutilizing a different weight, in parallel with one another instead ofserially as illustrated for ease in FIG. 3 .

When the decision at block 366 is no, the system proceeds to block 368,where the system evaluates the trained models and selects a subset ofthe trained models based on the evaluation. The system can evaluate eachof the models in view of metric(s) that seek to evaluate whether themodel effectively generates predicted measures that discriminate betweenhigh quality and/or high popularity user accounts and low quality and/orlow popularity user accounts. For example, the system can generate, fora trained model, both a quality metric and a popularity metric. Forinstance, the system can generate the quality metric based on how manyknown low quality user accounts, when their corresponding values areprocessed using the trained model, result in predicted measures thatfail to satisfy a threshold (with higher quantities indicating the modelis effectively filtering out low quality user accounts). Also, forinstance, the system can generate the popularity metric based on howmany known highly popular user accounts, when their corresponding valuesare processed using the trained model, result in predicted measures thatsatisfy the threshold (with higher quantities indicating the model iseffectively including highly popular user accounts). The system can thendetermine whether to deploy the trained model based on the qualitymetric and the popularity metric. As one example, if the quality metricof the trained model is greater than or equal to the quality metric ofan already deployed model, and the popularity metric of the trainedmodel is greater than the popularity metric of the already deployedmodel, the trained model can replace the already deployed model. Asanother example, if the quality metric of the trained model satisfies athreshold, and the popularity metric of the trained model is greaterthan the popularity metric of any other trained model, then the trainedmodel can be selected over the other trained model(s).

The system then proceeds to block 370 and deploys the model(s), selectedat block 368, for use.

FIG. 4 is a flowchart illustrating a method 400 of an example method ofusing a trained model in generating user account measures for useraccounts. For convenience, the operations of the flow chart aredescribed with reference to a system that performs the operations. Thissystem may include one or more components, such as one or more computingsystems that implement the user account measure engine 162 of FIG. 1 .While operations of method 400 are shown in a particular order, this isnot meant to be limiting. One or more operations may be reordered,omitted or added.

At block 452, the system identifies a user account of an onlineplatform.

At block 454, the system determines values, for defined features, thatare specific to the user account identified at block 452. The values forthe defined features are identified based on the defined features beingdefined for the trained model to be utilized at block 456.

At block 456, the system processes the values determined at block 454,using a trained model (e.g., trained based on method 300 of FIG. 3 ), togenerate a user account measure for the user account.

At block 458, the system stores an association of the user accountmeasure to the user account. For example, the association can be apointer or other data structure that associates the user account withthe user account measure.

FIG. 5 is a flowchart illustrating a method 500 of using user accountmeasures for user accounts. For convenience, the operations of the flowchart are described with reference to a system that performs theoperations. This system may include one or more components, such as oneor more computing devices implementing the threshold engine 164, thecontent retrieval engine 166, and/or the output engine 168 of FIG. 1 .While operations of method 500 are shown in a particular order, this isnot meant to be limiting. One or more operations may be reordered,omitted or added.

At block 552, the system identifies at least one query. Block 552optionally includes sub-block 553A or sub-block 553B. At sub-block 553A,the system identifies a query based on its submission by client device(e.g., submission at a search interface responsive to user interfaceinput). At sub-block 553B, the system identifies the query based ondetermining it is trending and/or is otherwise relevant to at least aplurality of users of client devices.

At block 554, the system determines threshold(s) based on one or moreproperties of the query identified at block 552. The threshold(s) can beused in sub-blocks 557A and/or 557B (described below). In otherimplementations, the system can instead use a default and/or staticthreshold(s).

At block 556, the system retrieves content items of online platform(s)based on the content items being responsive to the query. For example,the content items can be retrieved based on their underlying contentmatching one or more (e.g., all) terms of a text-based query. Block 556optionally includes sub-block 557A and/or sub-block 557B.

At sub-block 557A, the system restricts the search corpus to useraccounts with measures that satisfy the threshold of block 554. Forexample, the system can restrict the search to only content items fromthose user account(s) whose user account measures (e.g., as determinedbased on method 400 of FIG. 4 ) satisfy the threshold.

At sub-block 557B, the system filters content items to remove those fromuser accounts that fail to satisfy a threshold. When sub-block 557A andsub-block 557B are both performed, the threshold used in sub-block 557Bcan be the same as that used in sub-block 557A or, optionally, a morerestrictive threshold than that utilized in sub-block 557A.

At block 558, the system causes one or more of the content items,retrieved and not filtered out, to be rendered at client device(s). Forexample, the system can transmit the content item(s) for rendering atthe client device in a search result(s) page, a breaking news webpage,or a stand-alone push notification. Block 558 optionally includessub-block 559, in which the system causes the content items to berendered in dependence on the measure for their user accounts and/orbased on query-dependent measure(s) and/or content specific measure(s).For example, the system can determine an order of presentation of thecontent items, or which content items are shown first (e.g., in acarousel), based at least in part (or solely) on the user accountmeasures for the user accounts for the content items. The system cantransmit the content items such that they are caused to be rendered atthe client device in such manner(s). In some implementations, the systemcan additionally or alternatively determine the presentation orderand/or which content items are rendered first based on query-dependentand/or content specific measures as described herein.

Turning now to FIG. 6A and FIG. 6B, example graphical user interfaces600A and 600B for presenting, at a client device, content items that areselected based at least in part on user account measures. The graphicaluser interfaces 600A and 600B may be presented at one of the clientdevices 114 of FIG. 1 (e.g., in a browser executing at a client deviceand/or in another application executing at a client device) in responseto a transmission to one of the client devices 106 by the provisioningsystem 160 of FIG. 1 .

FIG. 6A illustrates an example of a content item 682A1 from a post and acontent item 682A2 from another post, that each include a “snippet” ofinformation from the corresponding post, and a link (e.g., theunderlined text may be a hyperlink) to the entirety of the correspondingpost. A user can select, via user interface input, a corresponding linkto cause the client device to navigate to the corresponding post. Thecontent item 682A1 and/or the content item 682A2 can be provided as apush notification (e.g., without requiring the user submit a query) toproactively notify the user of a hypothetical event in a hypotheticaltown. The content items 682A1 and 682A2 may have been identified basedon the illustrated query of “Hypothetical Event in Hypothetical Town”,which may be a trending query that is relevant to a user of the clientdevice (e.g., relevance determined based on user preferences and/or alocation of the client device). The content items 682A1 and 682A2 can beretrieved and/or caused to be rendered based at least in part on useraccount measures as described herein. For example, content item 682A1can be one selected for providing based on a user account measure for“Jon Doe” and content item 682A2 can be one selected for providing basedon a user account measure for “Jane Roe”. Content from additional postsmay also be provided, as indicated by the ellipsis in FIG. 6A.

FIG. 6B illustrates content items 682B1 and 682B2, from a post andcontent 682B2 from another post. In FIG. 6B, the content item 682B1 isthe same as content item 682A1 of FIG. 6A and the content item 682B2 isthe same as content item 682A2 of FIG. 6A. However, the content items682B1 and 682B2 are displayed differently, and are presented as part ofa carousel of content items. Further, in FIG. 6B the user has searched,in search bar 681B, for “hypothetical event”, and the content items682B1 and 682B2 are retrieved and/or provided responsive to that search(instead of proactively in FIG. 6A).

Although examples of graphical interfaces are presented in FIGS. 6A and6B, it is understood that alternative forms of presenting (audiblyand/or graphically) content items may additionally or alternatively beutilized.

FIG. 7 is a block diagram of an example computing device 710 that mayoptionally be utilized to perform one or more aspects of techniquesdescribed herein. Computing device 710 includes at least one processor714 (e.g., a CPU, GPU, and/or TPU) which communicates with a number ofperipheral devices via bus subsystem 712. These peripheral devices mayinclude a storage subsystem 724, including, for example, a memorysubsystem 725 and a file storage subsystem 726, user interface outputdevices 720, user interface input devices 722, and a network interfacesubsystem 715. The input and output devices allow user interaction withcomputing device 710. Network interface subsystem 715 provides aninterface to outside networks and is coupled to corresponding interfacedevices in other computing devices.

User interface input devices 722 may include a keyboard, pointingdevices such as a mouse, trackball, touchpad, or graphics tablet, ascanner, a touchscreen incorporated into the display, audio inputdevices such as voice recognition systems, microphones, and/or othertypes of input devices. In general, use of the term “input device” isintended to include all possible types of devices and ways to inputinformation into computing device 710 or onto a communication network.

User interface output devices 720 may include a display subsystem, aprinter, a fax machine, or non-visual displays such as audio outputdevices. The display subsystem may include a cathode ray tube (CRT), aflat-panel device such as a liquid crystal display (LCD), a projectiondevice, or some other mechanism for creating a regular image. Thedisplay subsystem may also provide non-visual display such as via audiooutput devices. In general, use of the term “output device” is intendedto include all possible types of devices and ways to output informationfrom computing device 710 to the user or to another machine or computingdevice.

Storage subsystem 724 stores programming and data constructs thatprovide the functionality of some or all of the modules describedherein. For example, the storage subsystem 724 may include the logic toperform selected aspects of the methods described herein.

These software modules are generally executed by processor 714 alone orin combination with other processors. Memory 725 used in the storagesubsystem 724 can include a number of memories including a main randomaccess memory (RAM) 730 for storage of instructions and data duringprogram execution and a read only memory (ROM) 732 in which fixedinstructions are stored. A file storage subsystem 726 can providepersistent storage for program and data files, and may include a harddisk drive, a solid state drive, a floppy disk drive along withassociated removable media, a CD-ROM drive, an optical drive, orremovable media cartridges. The modules implementing the functionalityof certain implementations may be stored by file storage subsystem 726in the storage subsystem 724, or in other machines accessible by theprocessor(s) 714.

Bus subsystem 712 provides a mechanism for letting the variouscomponents and subsystems of computing device 710 communicate with eachother as intended. Although bus subsystem 712 is shown schematically asa single bus, alternative implementations of the bus subsystem may usemultiple busses.

Computing device 710 can be of varying types including a workstation,server, computing cluster, blade server, server farm, or any other dataprocessing system or computing device. Due to the ever-changing natureof computers and networks, the description of computing device 710depicted in FIG. 7 is intended only as a specific example for purposesof illustrating some implementations. Many other configurations ofcomputing device 710 are possible having more or fewer components thanthe computing device depicted in FIG. 7 .

In situations in which the systems described herein collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures collect user information (e.g., information about a user'ssocial network, social actions or activities, profession, a user'spreferences, or a user's current geographic location), or to controlwhether and/or how to receive content from the content server that maybe more relevant to the user. Also, certain data may be treated in oneor more ways before it is stored or used, so that personal identifiableinformation is removed. For example, a user's identity may be treated sothat no personal identifiable information can be determined for theuser, or a user's geographic location may be generalized wheregeographic location information is obtained (such as to a city, ZIPcode, or state level), so that a particular geographic location of auser cannot be determined. Thus, the user may have control over howinformation is collected about the user and/or used.

In some implementations, a method implemented by one or more processorsis provided that includes, for each of a plurality of identified useraccounts: generating a corresponding quality measure; generating acorresponding popularity measure; and identifying a plurality ofcorresponding values for defined features, where the correspondingvalues are specific to the user account. The method further includesgenerating a plurality of training instances. The training instanceseach include: input that includes the corresponding values for acorresponding one of the user accounts, and labeled outputs of thecorresponding quality measure and the corresponding popularity measure,for the corresponding one of the user accounts. The method furtherincludes training a model using the training instances. The model can beused, once trained, to generate predicted measures, for user accounts,by processing the defined features. Training the model includes, foreach of the training instances: processing, using the model, the inputof the training instance to generate a corresponding predicted measure,and generating a corresponding loss based on evaluating thecorresponding predicted measure based on the labeled outputs of thetraining instance, including both the corresponding quality measure andthe corresponding popularity measure. The method further includesupdating the model based on the corresponding losses.

These and other implementations may include one or more of the followingfeatures.

In some implementations, generating the corresponding loss includes:determining a first error based on comparing the corresponding predictedmeasure to the corresponding quality measure; determining a second errorbased on comparing the corresponding predicted measure to thecorresponding popularity measure; and determining the loss as a functionof the first error and the second error. In some of thoseimplementations, determining the loss as the function of the first errorand the second error includes applying a first weighting to the firsterror and a second weighting to the second error. For example, the firstweighting can be a weight and the second error can be one minus theweight. In some versions of those implementations, the first error andthe second error are both mean squared errors and/or the method furtherincludes generating the weight using a black-box optimizer. In someadditional and/or alternative versions of those implementations, themethod further includes training, using the training instances, anadditional model to generate predicted measures by processing thedefined features. Training the additional model can include: using analternate weight in lieu of the weight; evaluating the model subsequentto training the model; evaluating the additional model subsequent totraining the additional model; and deploying, based on the evaluating,the better-performing of the model and the additional model.

In some implementations, generating the corresponding quality measurefor each of the plurality of user accounts includes: identifying anonline account page for the user account; providing the online accountpage for a plurality of discrete evaluations; receiving the plurality ofdiscrete evaluations; and generating the corresponding quality measureas a function of the discrete evaluations. In some of thoseimplementations, generating the corresponding quality measure as afunction of the discrete evaluations includes selecting a minimum of thediscrete evaluations, or a next-to-minimum of the discrete evaluations,as the corresponding quality measure.

In some implementations, generating the corresponding popularity measurefor each of the plurality of user accounts includes: identifying anonline account page for the user account; determining, from historicalrecords, a quantity of user interactions with the online account page;and generating the corresponding popularity measure as a function of thequantity of user interactions.

In some of those implementations, generating the correspondingpopularity measure as the function of the discrete evaluations includesgenerating the corresponding quality measure as the minimum of: (1) thelogarithm of the quantity; and (2) one or other fixed value.

In some implementations, the defined features include one or more of: aPagerank of an online account page for the user account; a status of theuser account, that is assigned by an online platform for the useraccount; a quantity of user interactions with the online account page; asentiment measure that is based on content items that are generated bythe user account; a quantity of links, to the online account page, fromdomains that are in addition to online platform domains associated withthe online platform for the user account; a link quality measure that isbased on links included in content items that are generated by the useraccount; or a primary language for the user account.

In some implementations, the model is a regression model.

In some implementations, the method further includes, subsequent totraining the model: identifying an additional user account of the atleast one platform; determining, for the additional user account, aplurality of additional corresponding values for the defined features,where the additional corresponding values are specific to the additionaluser account; processing the plurality of additional values, using themodel, to generate a predicted measure for the additional user account.Optionally, the predicted measure is output (e.g., transmitted) and/orthe method further includes determining, based on the predicted measure,whether to render, responsive to a query, a content item that isresponsive to the query and that is generated by the user account. Insome versions of those implementations, the predicted measure isdetermined prior to submission of the query and determining, based onthe predicted measure, whether to render, responsive to the query, acontent item that is responsive to the query and that is generated bythe user account, includes: determining, based on the predicted measure,to restrict the content item that is responsive to the query and that isgenerated by the user account; and in response to determining torestrict the content item, restricting the content item that isresponsive to the query and that is generated by the user account. Insome of those versions, restricting the content item that is responsiveto the query and that is generated by the user account includes:preventing searching, for the query, of any content item generated bythe user account; and/or filtering, from responsive content items fromsearching for the query, of any content items generated by the useraccount. In some additional or alternative versions of thoseimplementations, determining, based on the predicted measure, whether torender, responsive to the query, the content item that is responsive tothe query and that is generated by the user account, includes comparingthe predicted measure to a threshold, and determining whether to renderthe content item based on whether the predicted measure satisfies thethreshold. In some of those additional or alternative versions, themethod further includes determining the threshold based on one or moreproperties of the query, such as a primary classification of the queryand/or a geographical extent of trending of the query.

In some implementations, a method implemented by one or more processorsis provided that includes, for each of a plurality of user accounts:identifying a plurality of corresponding values for defined features,where the corresponding values are specific to the user account;processing the corresponding values, using a trained model, to generatea corresponding predicted measure for the user account; and assigning,in one or more computer readable media, an association of thecorresponding predicted measure to the user account. The method furtherincludes, subsequent to assigning the associations of the correspondingpredicted measures to the user accounts, determining, using thecorresponding predicted measures, to, for at least a given query,restrict any content item, generated by a first subset of the useraccounts, that is responsive to the one or more queries. Determining torestrict any content item generated by the first subset of the useraccounts is based on the corresponding measures, for the first subset ofthe user accounts, failing to satisfy a threshold. The method furtherincludes, in response to determining to restrict the content item,restricting rendering of any content item that is responsive to thegiven query.

These and other implementations may include one or more of the followingfeatures.

In some implementations, the method further includes determining, usingthe corresponding predicted measures, to, for the given query, render anadditional content item, generated by a given user account that is notin the first subset, that is responsive to the given query. In thoseimplementations, determining to render the additional content item isbased on the corresponding measure, for the given user account,satisfying a threshold. In some versions of those implementations, themethod further includes, in response to determining to render theadditional content item, transmitting the additional content item to aclient device to cause rendering of the additional content item. In someversions of those implementations, transmitting the additional contentitem to the client device is responsive to a submission, by the clientdevice, of the given query. In some other versions of thoseimplementations, transmitting the additional content item to the clientdevice is independent of submission of any query, including the givenquery, by the client device.

In some implementations, restricting any content item that is responsiveto the given query and that is generated by the first subset of the useraccounts includes: preventing searching, for the given query, of anycontent items generated by the first subset of the user accounts; and/orfiltering, from responsive content items from searching for the givenquery, of any content items generated by the first subset of the useraccounts.

In some implementations, the method further includes determining thethreshold based on one or more properties of the given query.

In some implementations, the trained model is trained based on lossesthat are each determined as a function of both a corresponding qualitymeasure label and a corresponding popularity measure label.

Various implementations disclosed herein may include one or morenon-transitory computer readable storage media storing instructionsexecutable by a processor (e.g., a central processing unit (CPU),graphics processing unit (GPU), and/or Tensor Processing Unit (TPU)) toperform a method such as one or more of the methods described herein.Yet other various implementations may include a system of one or morecomputers that include one or more processors operable to execute storedinstructions to perform a method such as one or more of the methodsdescribed herein.

What is claimed is:
 1. A method implemented by one or more processors,the method comprising: for each of a plurality of identified useraccounts: generating a corresponding quality measure; generating acorresponding popularity measure; identifying a plurality ofcorresponding values for defined features, wherein the correspondingvalues are specific to the user account; generating a plurality oftraining instances that each include: input that includes thecorresponding values for a corresponding one of the user accounts, andlabeled outputs of the corresponding quality measure and thecorresponding popularity measure, for the corresponding one of the useraccounts; training, using the training instances, a model to generatepredicted measures by processing the defined features, wherein trainingthe model comprises: for each of the training instances: processing,using the model, the input of the training instance to generate acorresponding predicted measure, and generating a corresponding lossbased on evaluating the corresponding predicted measure based on thelabeled outputs of the training instance, including both thecorresponding quality measure and the corresponding popularity measure;and updating the model based on the corresponding losses.
 2. The methodof claim 1, wherein generating the corresponding loss based onevaluating the corresponding predicted measure based on the labeledoutputs of the training instance, including both the correspondingquality measure and the corresponding popularity measure, comprises:determining a first error based on comparing the corresponding predictedmeasure to the corresponding quality measure; determining a second errorbased on comparing the corresponding predicted measure to thecorresponding popularity measure; and determining the loss as a functionof the first error and the second error.
 3. The method of claim 2,wherein determining the loss as the function of the first error and thesecond error comprises: applying a weight to the first error indetermining the loss; and applying one minus the weight to the seconderror in determining the loss.
 4. The method of claim 3, wherein thefirst error and the second error are both mean squared errors.
 5. Themethod of claim 3, further comprising: generating the weight using ablack-box optimizer.
 6. The method of claim 3, further comprising:training, using the training instances, an additional model to generatepredicted measures by processing the defined features, wherein trainingthe additional model comprises using an alternate weight in lieu of theweight; evaluating the model subsequent to training the model;evaluating the additional model subsequent to training the additionalmodel; and deploying, based on the evaluating, the better-performing ofthe model and the additional model.
 7. The method of claim 1, whereingenerating the corresponding quality measure for each of the pluralityof user accounts comprises: identifying an online account page for theuser account; providing the online account page for a plurality ofdiscrete evaluations; receiving the plurality of discrete evaluations;and generating the corresponding quality measure as a function of thediscrete evaluations.
 8. The method of claim 7, wherein generating thecorresponding quality measure as a function of the discrete evaluationscomprises: selecting a minimum of the discrete evaluations, or anext-to-minimum of the discrete evaluations, as the correspondingquality measure.
 9. The method of claim 1, wherein generating thecorresponding popularity measure for each of the plurality of useraccounts comprises: identifying an online account page for the useraccount; determining, from historical records, a quantity of userinteractions with the online account page; and generating thecorresponding popularity measure as a function of the quantity of userinteractions.
 10. The method of claim 9, wherein generating thecorresponding popularity measure as the function of the discreteevaluations comprises: generating the corresponding quality measure asthe minimum of: (1) the logarithm of the quantity; and (2) one or otherfixed value.
 11. The method of claim 1, wherein the defined featurescomprise one or more of: a Pagerank of an online account page for theuser account; a status of the user account, that is assigned by anonline platform for the user account; a quantity of user interactionswith the online account page; a sentiment measure that is based oncontent items that are generated by the user account; a quantity oflinks, to the online account page, from domains that are in addition toonline platform domains associated with the online platform for the useraccount; a link quality measure that is based on links included incontent items that are generated by the user account; or a primarylanguage for the user account.
 12. The method of claim 1, wherein themodel is a regression model.
 13. The method of claim 1, furthercomprising, subsequent to training the model: identifying an additionaluser account of the at least one platform; determining, for theadditional user account, a plurality of additional corresponding valuesfor the defined features, wherein the additional corresponding valuesare specific to the additional user account; processing the plurality ofadditional values, using the model, to generate a predicted measure forthe additional user account; and determining, based on the predictedmeasure, whether to render, responsive to a query, a content item thatis responsive to the query and that is generated by the additional useraccount.
 14. The method of claim 13, wherein the predicted measure isdetermined prior to submission of the query and wherein determining,based on the predicted measure, whether to render, responsive to thequery, a content item that is responsive to the query and that isgenerated by the user account, comprises: determining, based on thepredicted measure, to restrict the content item that is responsive tothe query and that is generated by the additional user account; and inresponse to determining to restrict the content item, restricting thecontent item that is responsive to the query and that is generated bythe additional user account.
 15. The method of claim 14, whereinrestricting the content item that is responsive to the query and that isgenerated by the additional user account comprises: preventingsearching, for the query, of any content item generated by theadditional user account; or filtering, from responsive content itemsfrom searching for the query, of any content items generated by theadditional user account.
 16. The method of claim 13, whereindetermining, based on the predicted measure, whether to render,responsive to the query, the content item that is responsive to thequery and that is generated by the additional user account, comprises:comparing the predicted measure to a threshold, and determining whetherto render the content item based on whether the predicted measuresatisfies the threshold.
 17. The method of claim 16, further comprising:determining the threshold based on one or more properties of the query.18. The method of claim 17, wherein the one or more properties of thequery comprise: a primary classification of the query; and/or ageographical extent of trending of the query.
 19. A method implementedby one or more processors, comprising: for each of a plurality of useraccounts: identifying a plurality of corresponding values for definedfeatures, wherein the corresponding values are specific to the useraccount; processing the corresponding values, using a trained model, togenerate a corresponding predicted measure for the user account; andassigning, in one or more computer readable media, an association of thecorresponding predicted measure to the user account; subsequent toassigning the associations of the corresponding predicted measures tothe user accounts: determining, using the corresponding predictedmeasures, to, for at least a given query, restrict any content item,generated by a first subset of the user accounts, that is responsive tothe given query, wherein determining to restrict any content itemgenerated by the first subset of the user accounts is based on thecorresponding measures, for the first subset of the user accounts,failing to satisfy a threshold; and in response to determining torestrict the content item, restricting rendering of any content itemthat is responsive to the given query.
 20. The method of claim 19,further comprising: determining, using the corresponding predictedmeasures, to, for the given query, render an additional content item,generated by a given user account that is not in the first subset, thatis responsive to the given query, wherein determining to render theadditional content item is based on the corresponding measure, for thegiven user account, satisfying a threshold.